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October 2015 


The Legislative Audit Committee 
of the Montana State Legislature: 


This is our information systems audit of the Combined Healthcare Information and 
Montana Eligibility System—Enterprise Architecture (CHIMES-EA) managed by the 
Human and Community Services Division (HCSD) in the Department of Public 
Health and Human Services (department). 


This report provides the Legislature information about system management processes 
and system eligibility and benefit calculations for the Supplemental Nutrition Assistance 
Program (SNAP) and Temporary Assistance for Needy Families (TANF). ‘This report 
includes recommendations for increasing oversight of management processes and the 
contractor, and improving documentation, monitoring, and communication to ensure 


the system is functioning as intended and effectively. 


We wish to express our appreciation to department personnel for their cooperation 
and assistance during the audit. 


Respectfully submitted, 
/s/ Tori Hunthausen 


Tori Hunthausen, CPA 
Legislative Auditor 
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REPORT SUMMARY 


The CHIMES-EA system is largely functioning as expected with regard to 
client analysis and the issuance of benefits. However, the department needs 
to make improvements 1n several areas to increase the system’s accuracy and 
efficiency, and improve user perception of system functionality. These include 
strengthening a number of processes for monitoring system performance; 
improving the process to repair system defects; strengthening the review of 
certain benefit overpayments; improving communication; and strengthening 


review of user access to the system. 


Context 


The Human and Community Service Division 
(HCSD), within the Department of Public 
Health and Human Services, manages public 
assistance programs for the state of Montana. 
The programs include: 


¢ Supplemental Nutrition Assistance 


Program (SNAP) 


¢ ‘Temporary Assistance for Needy 
Families (TANF) 


¢ Medicaid Eligibility 
¢ Healthy Montana Kids Eligibility 


The Combined Health Information and 
Montana Eligibility System - Enterprise 
Architecture (CHIMES-EA) was implemented 
in November 2012 as the system to manage 
the eligibility for these programs. Applicant 
data is entered once and the system executes 
multiple, complex rules for each program to 
determine eligibility and benefit amount. Since 
implementation, the system has been the focus 
of legislative committee meetings and reports. 


Due to the complexity of the system and 
inherent risk of these public assistance 
programs, our audit was purposed with 


reviewing processes in place to ensure the 
system is functioning according to policy 
and regulations and monitored for efficiency. 
We also reviewed processes that address and 
resolve system issues and monitor user access, 
as well as conducted a survey of system users. 
Without a properly functioning system, a 
process to monitor how the system is impacting 
the programs, or strong issue management 
and user access practices, data integrity and 
accuracy risks are increased. 


Results 


Audit work identified established processes 
for program metric review, but a lack of 
connection between program monitoring 
and system errors that could be influencing 
those metrics. Processes to identify and 
track root causes need to be strengthened to 
better understand how the system impacts 
the accuracy and timeliness of eligibility 
determinations and benefit calculations. 


The survey of system users provided valuable 
information for the audit and pointed to 
a poor perception of system functionality, 
which contrasted department management's 


(continued on back) 


perception. Survey results were used 
throughout the report to support audit 
findings. 


Audit testing and observations found errors 
within the system; however a process for issue 
management exists to address them. Audit 


work noted processes related to governance, 
prioritization of issues, addressing root cause. 


and documentation lacked controls to ensure 
effectiveness. The issue management process 
is not monitored effectively to ensure timely 
resolution of issues or that it meets the needs 
of users and the department. 


Throughout the audit, documentation of 


the system, policy, and procedures involving 


system management were noted as incomplete, 


out-of-date, or did not exist. Processes to 
ensure completeness and accuracy and review 
this documentation need to be strengthened. 


Audit work noted a process for granting, 


updating, and reviewing access of individuals. 
However, strengthening the review of access 
to system databases and privileged functions 
and monitoring activity will increase data 
integrity for the program. 


m{cvexovanlanteyalerelice)amexe)arelelatsyaler> 


Source: Agency audit response included in 
final report. 





For a complete copy of the report (1I5DP-01) or for further information, contact the 
Legislative Audit Division at 406-444-3122; e-mail to lad@mt.gov; or check the web site at 


http://leg.mt.gov/audit 
Report Fraud, Waste, and Abuse to the Legislative Auditor’s FRAUD HOTLINE 


Call toll-free 1-800-222-4446, or e-mail ladhotline@mt.gov. 





Chapter | — Introduction 


Introduction 


The Human and Community Services Division (HCSD), within the Department of 
Public Health and Human Services (department), manages public assistance programs 


for the state of Montana. The programs include: 
¢ Supplemental Nutrition Assistance Program (SNAP) 
¢ Temporary Assistance for Needy Families (TANF) 
¢ Medicaid Eligibility 
¢ Healthy Montana Kids (HMK) Eligibility 


The Combined Healthcare Information and Montana Eligibility System — Enterprise 
Architecture (CHIMES-EA) is the system that manages eligibility and issues benefits 
for these programs for over 125,000 clients across the state. The system allows for 
various aspects of client data to be entered in a central location and then determines 
eligibility for all programs to streamline the process and issue benefits as necessary. It 
manages these processes and assists in other daily tasks for almost 1,000 users across 
the state. 


The department relies on a contractor for development and maintenance of 
CHIMES-EA. ‘The contractor was hired in 2010 for development, implemented 
CHIMES-EA in November 2012, and has been maintaining the system since then. 
Since implementation, the system has been the focus of legislative committee meetings, 
reviews, and reports. This, combined with system complexity and an inherent risk 


within public assistance programs, warranted an audit of how the system is functioning. 


Background 


Prior to the current system, The Economic Assistance Management System (TEAMS) 
was used for determining eligibility for each program and required a more manual 
process. The implementation of CHIMES-EA added automated benefit calculation 
through a new rules engine in a single, web-based system. ‘The rules engine contains 
and executes the business rules for each public assistance program. In addition, 
functionality to support fiscal operations required by SNAP and TANF was added. 
This functionality, known as the Shared Fiscal Services Layer, is a single financial 
application that communicates payment information to the Statewide Accounting, 
Budgeting, and Human Resources System (SABHRS) to issue benefits. CHIMES-EA 
does not issue money to clients. Medicaid and HMK eligibility had been managed ina 
separate system, but were recently added to CHIMES-EA in spring of 2015. 
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Audit Scope and Objectives 

Our audit focused on CHIMES-EA functionality for SNAP and TANF. During 
our audit, the Medicaid and HMK eligibility programs were being migrated into 
the CHIMES-EA system from the previous system, so its functionality was not 
reviewed. However, many of the system processes we reviewed are shared across all 
public assistance programs, so our testing has applicability to the Medicaid portion of 
CHIMES-EA. We also reviewed controls for issue management and user access and 


monitoring of the entire system. 


We developed four audit objectives, listed as follows: 
1. Determine if the agency is effectively monitoring the system. 


2. Determine if business rules are executed within the system according to 
policy. 
3. Determine if controls exist to ensure the effectiveness of issue management. 


Determine if controls exist to ensure data integrity. 


Methodology 
CHIMES-EA is not a fully-automated means of determining eligibility and issuing 


benefits for public assistance programs; the system stores and processes information, 
but eligibility determinations and benefit calculations have to be reviewed by an 
individual before being authorized. Because of this, there is both a system control, 
the automatic eligibility determination and benefit calculation, and a manual control, 
the person reviewing the case. We reviewed how the system functions and how the 


department controls the system to ensure benefits are as accurate as possible. 


Methodology included the following: 


User Survey: We conducted a survey of users with access to CHIMES-EA. The survey 
provided information that will be addressed throughout the report. ‘The list of survey 
questions and summary responses can be found in Appendix A. ‘The survey was sent 
to 894 users with 512 users responding. These users represent various roles that work 


with the system including: 
¢ Office of Public Assistance (OPA) 
¢ Work Readiness Component (WoRC) 
¢ = Child Support and Enforcement Division (CSED) 
¢ = Internal Audit 


¢ Management and Central Office 


Interviews: Discussion with various users and managers of the system. Individual 


staff include: 
¢ Central Office 
¢ Project Management Bureau 
¢ OPA 
* ‘CSED 
© 


Quality Assurance Division 


Department of Labor and Industry 


Site Observations: We visited four OPA offices to discuss the system with users and 


observe how users interact with it on a daily basis. OPAs visited were: 


o 


o 


® 


o 


Helena 
Missoula 
Billings 


Butte 


Comparison to Industry Standards: We compared various processes to industry 


standards. Industry standards used include: 


o 


Control Objectives for Information and Related Technology (COBIT) — 
Standards for Information Technology (IT) management and governance 
based on the consolidation of more than 50 IT good practice sources 
published by various international standards bodies, governments, and other 
institutions. These standards outline control practices to reduce technical 
issues and business risks. 


National Institute of Standards and Technology (NIST) — Provides a catalog 
of security and privacy controls for information systems. Montana State 
policy requires the use of NIST as guidance for security risk management 
and has established baseline security controls from NIST. 


Information Technology Infrastructure Library (ITIL) — A set of standard 
practices that focuses on aligning IT services with the needs of the 
organization and its users. 


System Testing and Observation: Within a test system of CHIMES-EA, we created 


applications, entered test information, and evaluated how the system responded. We 


also viewed data and cases within the actual system, but did not have access to enter, 


change, Of process any Cases. 
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Reports: Reports were reviewed to provide information relating to the audit objectives. 


Report information included: 
¢ Issue Management 
¢ User and Contractor Access 
¢ Overpayments 


¢ System Testing 


Documentation Review: Our audit reviewed documentation depicting system 

design, expected functionality, and processes involved in managing the system. 
Documentation included: 

¢ Business Rules: For our audit, we judgmentally selected and reviewed 

business rules related to two areas: how eligibility and benefit calculations 

are determined and how correspondence is triggered to be sent to clients. 


Specific areas for eligibility and benefit calculations included nonfinancial 
decisions, authorizations, and business rules to calculate benefit accounts. 


¢ Interfaces: Detailed design documents that define when interfaces are 
supposed to occur, what information is transferred and the format, and how 
information will be monitored to ensure the interface is working. 


¢ Benefit Discrepancies: Documentation of the process to manage discrepancies 
and how the system determines a discrepancy. 


¢ Issue Management: Documentation of the process and key points within 
the process including: testing, department approvals, and issue classification. 


¢ Department Procedures: Documentation for procedures within the 
department to manage the system, including policy and user guides for the 
system, and user access procedures. 


Management Memorandum 


A management memorandum is a verbal or written notification to the agency for 
issues that should be considered by management, but do not require a formal agency 
response. We issued a management memorandum to the department regarding 
development of a plan to convert data still used within the previous system, TEAMS, 


into a sustainable, cost-effective source. 


Overall Conclusion 


The CHIMES-EA system is largely functioning as expected with regard to client 
analysis and the issuance of benefits. However, the department needs to make 
improvements in several areas to increase the system’s accuracy and efficiency. 
These include strengthening a number of processes for monitoring, reviewing, and 
documenting system performance; improving the identification, prioritization, and 


repair of system defects; developing a process to review certain benefit overpayments; 


improving employee training and communication; and strengthening review of user 


access to the system. ‘This report addresses our findings in the following chapters: 


o 


Sd 


o 


Chapter IT - Performance Monitoring 
Chapter III - System Functionality 
Chapter IV - Issue Management 
Chapter V - User Access and Monitoring 


Chapter VI - Documentation 


Chapter II - Performance Monitoring 


Introduction 


Performance monitoring 1s the process of supervising progress to ensure an 
organization is on-course and on-schedule in meeting objectives and performance 
targets and metrics. This includes application monitoring and program monitoring. 
Application (system) monitoring is a process that ensures that a system performs in an 
expected manner. Program monitoring does the same, but ensures that the program, 
in this case public assistance, is performing in an expected manner. The Combined 
Healthcare Information and Montana Eligibility System — Enterprise Architecture 
(CHIMES-EA) is used to support the public assistance programs, so how the 
system performs affects the performance of the programs. Our first audit objective 
was to determine whether the Department of Public Health and Human Services 
(department) is effectively monitoring the performance of the CHIMES-EA system, 


which includes understanding how the system impacts program metrics. 


Industry standards indicate a strong performance monitoring process includes: 
¢ Specific goals for related metrics 


¢ Continuous, consistent, and timely review and communication of those 
metrics 


¢ Root cause analysis of identified issues 


Strong performance monitoring practices help ensure a system is operating as intended 
and establish performance baselines to be used in future planning. They also provide 
reliable information to understand how the system is affecting program metrics. A 
metric is a quantifiable measure that is used to track and assess the status of a specific 


process. Metrics are used to drive improvements. 


Performance Monitoring Processes 


Metrics used to determine the success of the public assistance programs are: 


¢ Timeliness: Benefit application processing time and number of pending 
applications. 


¢ Accuracy: Benefit amount and number of overpayments and underpayments. 


While specific goals have not been established, there are processes and tools in place to 


monitor timeliness and accuracy of the program. 


¢ Supplemental Nutrition Assistance Program (SNAP) Quality Control 
Review: ‘This is a federally mandated review completed by a separate 
division within the department. Independent reviews are conducted based 
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on a federally approved sample of SNAP cases that were approved or denied 
for benefits. These reviews are done monthly and sent to the federal Food and 
Nutrition Service (FNS) for a secondary review. This information is used for 
federal grading of states’ SNAP programs based on timeliness and accuracy. 


¢ Temporary Assistance for Needy Families (TANF) Quality Control 
Review: Department policy describes a review of TANF cases similar to 
that of SNAP. There has been no federal requirement for this type of review 
since 1996 when TANF was created; however, the department has a process 
for internal review of cases. This process includes randomly selecting cases 
for each Office of Public Assistance (OPA) manager to review monthly. The 


review includes all public assistance programs within the case. 


¢ Business Intelligence Tool: This tool was developed to provide various 
metrics and reports to help manage SNAP and TANF. These reports 
include drill-down capabilities for specific information as well as macro-level 
classification like region, office, and time frame. The department also plans 
to implement a dashboard within this tool that will bring the key metrics 
for the program and system together to assist in decision making and 
monitoring. 


¢ Application Monitoring: The department receives various information 
about system performance and availability. This information covers web 
service response times and availability, database traffic, login requests, and 
disk space and memory usage. 


Perception of System Functionality 


Part of system monitoring is understanding if the system is working as intended 
(functionality). Aside from reviewing metrics, our audit work included a web-based 
survey where users and central office staff were asked to provide their perceptions 
of system functionality. During interviews with central office staff, senior officials 
indicated the system is working as intended. This position has been reiterated in 
hearings and committee meetings. They believe the majority of program issues are 


related to user workload and business processes, not the system. 


The majority of respondents provided a different opinion than central office staff 
in relation to system functionality, shown in Figure 1 (see page 9). The majority of 
respondents are also users who work in the system for more than five hours a day. Their 


comments indicate the system is contributing to program and workload issues. 


Figure 1 
User Survey Response: Functionality 


Do you feel the system is functioning as intended? 


m Yes = No = Not Sure 


Source: Compiled by the Legislative Audit Division from survey responses. 





We reviewed this information further to determine if this opinion was limited to users 
with knowledge of the previous system, decommissioned two years prior to the survey. 
When combining OPA eligibility roles and supervisor roles, perception worsens with 
higher tenure in (shown in Figure 2). However, people with a few years of experience 
in the old system and people with no experience in the old system also indicate the 
system is not functioning as intended. 


Figure 2 
User Survey Response: OPA Roles and Tenure 


Do you feel the system is functioning as intended? 


10+ years 
5-10 years 
3-5 years 
O-2 years 


m Yes = No = Not Sure 


Source: Compiled by the Legislative Audit Division from survey responses. 
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Results show respondents feel the system is inconsistent in how often it performs basic 
system requirements as well (shown in Figure 3). 


Figure 3 
User Survey Response: Basic System Intentions 


In your opinion, is CHIMES-EA . . .? 

m Never = Sometimes = Most of the Time m Always 

User Friendly 

Responding in a timely manner 

Easy to learn 

Processing applications quickly 

Accurate 

Simplifying Processes 


Reducing Errors 


Allowing you to spend the majority of your 
time on qualitative customer care 


Increasing the efficiency of collection, 
reporting, and analysis of client data 


Source: Compiled by the Legislative Audit Division from survey responses. 





This difference in perception is a good example of why performance monitoring is 
important and valuable to an organization. Performance monitoring provides reliable 
and timely data that can be used to evaluate system and program operations. This 
information can then be communicated to staff to align perceptions with reality. 


Improving Performance Monitoring Practices 


The department has processes in place to gather data and monitor the program and 
the system, but the data is general and does not provide details to help understand 
if the system is functioning as intended. In addition, processes are not consistently 
followed. ‘The department does not have established, specific goals for each program or 
the system as a whole. The only metrics indicating progress are federal SNAP program 


goals of accuracy and timeliness. The department has no specific goals for what these 
metrics should be. As shown in the following table, federally reported SNAP metrics 


for accuracy and timeliness have been declining in recent years. 


Table 1 
Federally Reported SNAP Metrics for Montana 





Metrics Timeliness Accuracy 





% of Cases Under- 
Years Processed Issuance 
Timely Error Rate 


2011 91.96% 0.82% 4.20% 5.02% 
2012 93.29% 0.64% 2.07% 2.11% 
2013 88.62% 1.43% 4.56% 6.00% 
2014 NA 2.25% 5.00% 7.25% 


Overpayment Combined 
Error Rate Error Rate 



































Source: Compiled by the Legislative Audit Division. 





These metrics are generated using a complex process with multiple aspects of a case 
reviewed and documented, such as the amount of the discrepancy, causes for the 
discrepancy, and nature of issues found even if there was no monetary impact. The 
review procedure is outlined in department policy to ensure it is conducted consistently 


and so that the information found can be used for improvements. 


Through audit work, we found the department relies on reports and information 
from these reviews. These reviews provide information to assist the department in 
understanding the error rates and timeliness reports. One of the metrics determined 
through the review is cause. According to the reports, the main cause for errors the 
past three years is that reported information from the client is disregarded or not 
applied (25-30 percent of cases). System error causes for the same time period were 


4-8 percent. 


Root Cause Is Not Being Identified 


Our review of this process and these reports showed that, while these are the 
documented causes, they are only the general causes for the errors, not the root cause. 
The question of why the information reported was disregarded or not applied, or if 
the system caused the delay, is not answered. This requires further analysis by the 
department to understand how the system is impacting these causes and metrics. 
Policy indicates review findings should be consolidated for secondary review by central 
office, but it is not happening consistently. Root cause is not being identified or tracked 


to understand if the system is or is not influencing the error rate. Policy also indicates 


rr, SDPO 


11 


12 | Montana Legislative Audit Division ‘ Legislative Audit Division 


findings of this secondary review be communicated through the field for training and 


accuracy improvements. 


Since there has not been a federal mandate to conduct TANF reviews, they have not 
been done consistently or according to policy. There was no clear understanding of 
how the TANF program is working and how the system is functioning to support 
the program. During the audit, the department implemented a new internal review 
process that is intended to have OPA managers review all programs for a sample of 
cases each month, including TANF. Review of this new process showed that root 


cause of errors, and if the cause relates to the system, is not identified or tracked. 


Because of these deficiencies, there are no metrics to prove the system is working as 
intended. Without this type of information, the disconnect in perceptions of system 
functionality will continue. While management is working to create a management 
dashboard of various metrics to understand the success of the programs and assist in 
decision making, there are other areas of the performance monitoring process that 
could be strengthened to increase its effectiveness. By increasing analysis and tracking 
if the system is related to root cause, the department will better understand how the 
system is functioning, be able to isolate and correct issues more efficiently, and will be 


more informed in future system planning. 


a 


RECOMMENDATION #1 


We recommend the Department of Public Health and Human Services 
improve performance monitoring processes by: 


A. — Defining specific goals relative to the system and each program 
managed by the system. 


B. Defining and documenting a process to consistently review all data 
relative to these goals. 


© 


Reviewing information gathered to identify, document, and track system 
relation to root cause, and communicating results to users. 


— ‘i Py 


Chapter III - System Functionality 


Introduction 


The Combined Healthcare Information and Montana Eligibility System — Enterprise 
Architecture (CHIMES-EA) is a complex system, integrating multiple business rules 
(decisions based on federal and state statutes and rules, and department policy) into 
one place to increase efficiency for the Department of Public Health and Human 


Services (department) and clients. Functions include: 

¢ — Eligibility determinations for: 
Q Supplemental Nutrition Assistance Program (SNAP) 
Q ‘Temporary Assistance for Needy Families (TANF) 
Q Medicaid and Healthy Montana Kids 

¢ Benefit calculations 

¢ Communication with other systems through interfaces 

¢ On-going case management 

¢ Staff workload management 


¢ Reporting 


Industry standards identify these types of functions as controls to ensure accuracy and 
integrity. As such, these functions need to be thoroughly understood and monitored to 


ensure they are working as intended and are effective. 


CHIMES-EA manages over 900 business rules that define how the system should 
make decisions based on data entered by a user. This is the main function and most 
complex task required by the system, hence the need for a rules engine. Due to the 
magnitude of this system and the amount of rules that have to be executed within each 
program, there are inherent risks related to ensuring system processing is accurate. 
Strong controls need to be in place to mitigate these risks and ensure the integrity of 


the system is maintained. 


Our audit reviewed three main areas related to system functionality. The following 


questions address the focus of work in each area: 
¢ — Is the system applying business rules accurately? 
¢ Are interfaces controlled to ensure efficiency and effectiveness? 


¢ How is the system used to control benefit discrepancies? 
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Our audit shows that processes involving system interfaces, benefit discrepancies, and 
addressing system issues when business rules are not demonstrated accurately need to 
be strengthened to improve system functionality. The following sections discuss our 


audit work and findings. 


System Application of Business Rules 


The first step in our review of system functionality was understanding if the system 
demonstrates business rules accurately. This was accomplished through our own 


testing, reviewing department testing procedures, and site observations. 


We accessed the department’s test environment and created our own scenarios to view 
how the system functioned. This process consisted of creating a scenario based on the 
policy or business rule we were testing and then inputting information to determine 
eligibility and benefit amount relative to the defined conditions. We then adjusted the 
scenario by changing information to see if the system acted accordingly for varying 


conditions. 


Since our own testing in a test environment was unable to review certain scenarios 
that required historical data, we determined that it was necessary to observe how users 
manage cases in production and witness specific system functions in a live environment. 
We visited Office of Public Assistance (OPA) locations to discuss system functionality, 
observe interaction and specific functions, and follow up with users on questions we 


had from our survey. 


Testing System Functionality 


Our test environment review focused on system demonstration of policy, not thoroughly 
testing the system functionality like testing conducted during development. We created 
14 households that covered multiple types of family members, ages, and various other 
conditions pointing directly related to policy and business rule documentation that we 
reviewed. Based on the results of tests with these households, we limited our testing to 


just eligibility determination and benefit calculation policy. 


Of the 14 households we created, 9 households were not functioning as expected due 
to various reasons. We identified instances where the system did not accurately execute 
policy. We also encountered inefficiencies in the process for entering information. 
CHIMES-EA was designed to lead users through a series of screens and multiple 
questions to enter client information and subsequently determine eligibility and 
calculate benefits. We identified scenarios where information that directly impacted 
the outcome of eligibility and benefit amount was not required by the system during 


data entry. This information may not be required for authorization of benefits, but 


directly impacts the accuracy of benefits. During our testing, we had to go back 


and enter this information which created the inefficiency. A few of our test cases are 


described in Table 2. 


Table 2 
Auditor Test Scenario Results 





Scenario 


Expectations 


Results 





Single parent with 2 children applying 
for TANF. Oldest child turns 18 in 
month of application. 


TANF benefits would be issued for 
household of 3 in application month and 
household of 2 for months after that. 


Initial month was incorrectly denying 
oldest child. 





Married couple with two children 
from other relationships (not siblings) 
applying for TANF. 


Step parent income should be included 
as unearned income in determination 
and benefit calculation. 


Adult income was not accurately 
calculated to determine TANF eligibility 
and benefit amount. 





Single parent with 2 children applying 
for SNAP and TANF. 


Child support should be included as 
unearned income in determination and 
benefit calculation. 


Child support was not accurately 
calculated in income to determining 
benefit amount in either program. 





Married couple with 4 children, dad's 
on strike, applying for SNAP and TANF. 


TANF eligibility should be denied due 
to strike. SNAP eligibility should only be 
allowed if the family was eligible prior to 
the strike. 


Multiple, unrequired fields had to be 
entered before the system would 
correctly determine eligibility based 
on policy. The single field asking if 
a person was on strike did not deny 
benefits. 





Married couple with 3 children applying 
for SNAP and TANF. 


If a rent expense is entered and all 
required fields are complete, rent will be 
included as an expense in determination 
and benefit calculation. 


Rent was not recognized in expenses 
if a single, nonmandatory field is blank. 
The system does not force a "yes/no 
answer. 





Self-employed single parent with 3 
children applying for SNAP and TANF. 








Self-employment income should 
be included as earned income in 
determination and benefit calculation. 





Self-employment income was not 
accurately calculated to determine 
benefit amount in either program. 








Source: Compiled by the Legislative Audit Division. 
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With errors noted in our initial testing, along with the need to review more complex 
cases with historical data, we decided to stop creating additional test scenarios and 
moved forward to site observations. These observations assisted in understanding if 
our experience was similar to that in the field and if the cases we created were rare 


occurrences. 


Our site observations supported our own testing and gave us insight into other issues. 
These observations also coincided with our survey responses related to trust of system 
calculations. The following diagram, Figure 4 (see page 16), indicates users are still 


manually calculating benefit amounts to verify system calculations. 
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Figure 4 
User Survey Response: Manual Budgets 


How often are you creating manual budgets to verify CHIMES budgets? 


= Always = Most of the time = Sometimes = Rarely m Never 


Source: Compiled by the Legislative Audit Division from survey responses. 





Our survey also asked how often extra steps within the system are required to complete 
a case action. These extra steps can be a variety of manual actions taken by the user if 
the system is not functioning as intended. Figure 5 shows a more unfavorable response 
to the amount of cases requiring these extra steps. 


Figure 5 
User Survey Response: Workarounds 


How often do cases require some kind of workaround due to a system issue? 


1% 


m= Always = Most of the time = Sometimes m Rarely m Never 


Source: Compiled by the Legislative Audit Division from survey responses. 





To better understand these graphs and the reasons for these results, we discussed issues 
and observed user interaction with the system. Table 3 (see page 17) indicates areas 
where policy was not executed accurately or the system was not functioning according 
to business rules from both our own testing and site observations. 


Table 3 
Areas with System Functionality Issues Found in Testing and Observations 


Number of 
Issues Found 





Issue Area Examples 





2 - Testing SNAP Able Bodied Adult Without Dependents 
Eligibility Determination program eligibility was not always restricted to 
1 - Site three months within 36 month period. 








3 - Testing SNAP not including TANF grant in unearned 
9 - Site income when TANF grant was overridden. 





Benefit Calculation 





4 - Testing Enrollment status is a required field for 
authorization but is not marked as such during data 
entry. The user has to go back to fill out the status 
2 - Site for each person before benefits can be authorized. 


Data Entry/ User Interaction 





1 - Testing SNAP Simplified report notices with incorrect 
1 - Site income information. 





Client Correspondence 











Source: Compiled by the Legislative Audit Division. 





System issues impact the program in various ways, from inaccuracies to inefficiencies. 
They increase the amount of time it takes to complete a case due to correcting the 
system error, and increase mistrust of the system. Manual changes and other additional 
work increase the likelihood of future calculation issues requiring additional manual 
changes. A specific instance of inefliciency is related to alerts created by the system to 
direct the worker on cases that need to be reviewed. We found erroneous and duplicate 
alerts, as well as alerts that reappeared after being deleted. System alert issues increase 
work by requiring users to separate actual alerts from erroneous alerts, and increases 


the likelihood alerts may be ignored altogether. 


Fixing System Defects Has Not Been Prioritized 


During our fieldwork, there were approximately 800 identified defects within the 
system. A defect is terminology used for an instance where the system is not functioning 
as intended. Around half of these defects are eligibility and benefit calculation related. 
Of the eligibility defects, the majority are attributed to the recent migration of Medicaid 
into the system, while 25 percent are attributed to SNAP, TANF, or all programs. Due 
to the complexity of the system and amount of business rules managed, the amount 
of issues may not seem large. However, our concern is with how these issues are being 
addressed and understanding why there are still frustrations with a system that was 


implemented over TWO years ago. 


Further review shows that in the system updates planned for 2015 (see Table 4 on 
page 18), twice as many enhancements are being addressed as defects. Enhancements 


are changes to the system that are not instances of incorrect system functionality. This 
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is disproportionate to the number of 






































defects and enhancements that currently Table 4 
are pending for the system. As of the end Planned : pores i for 2015 
as Of Via 
of July 2015, there were 883 unresolved y 
system defects and 391 pending Update Defects Enhancements 
enhancements. 
7.1.0 3 
: 7.2.0 
Enhancements are sometimes necessary, 
especially when needed for federal ia 
: ‘ 8.1.0 
regulation and policy changes, but 
allowing defects to go unfixed increases one 
risks for the program and _ reduces 10.0.0 
efficiency for the users. The department is Total 
tasked with managing risk and balancing | Source: Compiled by the Legislative Audit 


the priorities of addressing both defects Division. 





and enhancements. By ensuring system 

defects that affect eligibility determination and benefit calculations are addressed 
promptly, the department can start to reduce the risk of inaccuracies caused by the 
system and reduce the backlog of defects. ‘This will also reduce the risk that the issues 
are compounded when further enhancements or changes are made to the system or 


program. 
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RECOMMENDATION #2 


We recommend the Department of Public Health and Human Services 
develop a plan to promptly address outstanding system defects that impact 
eligibility determinations and benefit calculations. 


Es 


System Interfaces 


Another step in our review of system functionality was reviewing interfaces between 
CHIMES-EA and other systems. Interfaces refer to the transfer of specific data 
between two systems to create efficiency and accuracy. Our work addressed the 


following question: 


¢ Are interfaces controlled to ensure efficiency and effectiveness? 


To examine interfaces, we evaluated logs that monitor how interfaces are working. We 
also discussed interfaces with users, both within and outside of the department, to 


understand details of what processes require information from CHIMES-EA. Testing 


each interface was not feasible with the amount of interfaces and types of parties 
involved (state and federal). We identified 56 interface documents and reviewed 38 
that were within the scope of the audit. These 38 documents included the following 


types of interfaces: 
¢ Eighteen internal interfaces to other department systems 
¢ Two interfaces to Department of Labor and Industry 


¢ = One interface each to the Office of Public Instruction, Department of Justice, 
and Department of Revenue 


¢ Fourteen interfaces to federal systems 


Interface Monitoring 


Since testing interfaces was not feasible, we sampled interface logs from this year to 
review and understand what controls are in place to ensure interfaces are working 
and issues are identified. Interface logs contain information for each interface that can 
be used to track different metrics. Metrics could include the number of records sent 
through the interface, the number of records processed and rejected, or the number of 
records that matched a certain data set. The metrics vary depending on the purpose 
and nature of the interface, so documentation of these metrics is included in each 
interface’s design document. Interfaces that are real-time, meaning the data exchange 
happens when the user requests it, are not monitored in these logs. The logs are meant 
to monitor interfaces that are a transfer of large amounts of data at one time, and 


usually done outside of normal work hours. 


The process for interface monitoring is managed by the contractor. Contractor staff 
review interfaces daily and create various monitoring documents, including logs, 
documentation of any issues that occurred, and a summary. [hese documents are then 
stored in a system that is used to manage multiple project management processes. An 
email with the summary, noting either success or failure for each interface, is then sent 


to appropriate department staff. 


The interface logs provide useful information; however, we identified issues within our 
judgmental sample of 17 interface logs. Seven interfaces were not monitored in the 
log on the days documentation indicated they should have run. We also noted that 


personally identifiable client information was recorded in two interface logs. 


Of the seven interfaces that were not monitored, one was no longer in use. The 
department was unaware that personal client information was being recorded in the 
log. The logs are stored in an environment where users that have no business need for 
this information have access to the information. The department immediately removed 


this information from being logged when notified of the issue. 
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While monitoring responsibility is assigned to the contractor, the department is 
ultimately accountable for how the interfaces are working. The department needs 
to increase oversight and review of the monitoring process outside of the daily email 
sent by the contractor to ensure the control, in this case interface logs, is working 
effectively. Industry standards indicate strong oversight includes monitoring internal 
controls and evaluating effectiveness and efficiency. By increasing oversight of interface 
monitoring, the department will ensure the reliability of what is being reported by 
the contractor and reduce the risk of an ineffective control. This will also ensure 
information monitored in the interface logs is consistent with design documentation. 
Instances where data is being logged that is not needed, for example the personally 


identifiable information found in our review, will be identified as well. 
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RECOMMENDATION #3 


We recommend the Department of Public Health and Human Services 
improve interface monitoring by: 


A. — Increasing oversight and review of the monitoring process. 


B. Ensuring only necessary information is logged and only users with a 
business need can access logged information. 
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Interface Functionality 


During our review of interfaces and from our survey, we received varying statements 
on how interfaces were working. Survey responses varied from users that always use 
interfaces, to users who never use them or were not aware of or do not need to use the 
interface. Our survey and site observations also provided comments from users stating 
they do not use interfaces and would rather go directly to the source system to obtain 
or verify data. In addition, users state the information provided through interfaces is 
out of date, incorrect, or missing. Department management indicated all interfaces 


were working and usable within the system. 


From our review of interface logs, we were able to determine that monitored interfaces 
were operational and logging data transfers, but this does not indicate the interface is 
functioning as intended. We reviewed help desk tickets (identified issues) regarding 
interfaces. We found 23 issues related to 13 of the interfaces we reviewed. While there 
is a process for managing issues, discussed in Chapter IV, we wanted to understand 


how issues are addressed with users receiving CHIMES-EA information. 


In order to do this, we further reviewed the exchange of information between 
CHIMES-EA and the System for the Enforcement And Recovery of Child Support 
(SEARCHS). SEARCHS has the most interfaces with CHIMES-EA in regards to the 


following information: 
¢ Child Support amount 
¢ ‘TANF Grant amount and client information 
¢ Non-Cooperation with Child Support Enforcement Division (CSED) 
¢ Out-of-State Child Support Benefits 
¢ Referrals to CSED when a parent starts receiving TANF 


Manual Processes Are Being Used for SEARCHS 


In our review of SEARCHS we found there are manual processes in place to ensure 
proper management of cases due to interface issues, such as child support payment and 
TANF eligibility and payment information not being shared. According to CSED, 
the interfaces work some of the time; however, CSED staff are using manual processes 
more often than intended. Due to the number of issues with transferred information, 


the reason why the information did not transfer or was incorrect cannot always be 


researched by CSED staff. 


These issues ultimately increase the risk that a parent may not receive the appropriate 
grant amount when receiving child support and TANF at the same time, which is 
against department policy. A parent is entitled to the greater of either the TANF or 
child support payment. CSED withholds the child support payment and passes on only 
the necessary amount if the child support payment is greater than the TANF grant. 
If a parent received both types of assistance in an amount greater than allowed, an 
overpayment should be established by the Office of Public Assistance (OPA). However, 
if the reason the error occurred was determined to be a system interface issue, by policy, 
the overpayment is not recouped. Thus reducing the amount of funds CSED receives 
and the amount of TANF grant money available for other needy families. 


Industry guidelines suggest organizations maintain an understanding of the system 
and all users impacted by it by regularly meeting with business units, internal users, 
divisions, and other types of users to understand current business problems, process 
bottlenecks, or other constraints that need to be addressed. ‘The department is aware of 
the issues with interfaces to SEARCHS and has help desk tickets identified. They have 
also held a meeting recently with CSED staff and the contractor to discuss these types 
of issues in detail. However, these meetings are not consistent and communication of 


progress outside of these meetings could be increased. 
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Specific interface problems should be identified and managed through the issue 
management process, which is addressed in Chapter IV. However, we found 
communication and collaboration need to be increased. By increasing communication 
with users relying on interfaces of CHIMES-EA information, the department 
strengthens its understanding of the entire system and how it is affecting all users. ‘This 
increased awareness and communication, for both sides, increases trust in the system, 


efficiency of issue resolution, and builds relationships. 


ee 
FRRECOMMENDATION #4. 

We recommend the Department of Public Health and Human Services 
strengthen interface controls by consistently: 

A. Meeting with all stakeholders to document and review current issues. 


B. Communicating with and updating stakeholders throughout the process. 
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Benefit Discrepancies 


The third step in understanding system functionality addresses the following question: 


¢ How is the system used to control benefit discrepancies? 


There are circumstances that create a discrepancy in the benefit amount authorized to 
be issued within the system, like client error or user input error. Strong controls related 
to how the system addresses discrepancies and how user procedures involving system 
discrepancies need to be established and followed to ensure benefits are paid according 


to policy. 


Benefit discrepancies are considered either an overpayment or underpayment. If an 
underpayment occurs, a supplemental payment is issued within the system to make 
up the difference. If an overpayment occurs, a claim is established within the system 
and the recovery process is initiated. CHIMES-EA is designed to identify these 
discrepancies and suggest the correction for the user to authorize, but also allows for 
users to manually create supplemental payments and overpayment claims if the system 
does not identify them. 


To review the discrepancy process, we evaluated system documentation and policy and 


gained an understanding of how the system is involved. We also asked users how they 


feel the system was assisting in identifying benefit discrepancies. Survey results, shown 
in Figure 6, indicate that users are split on how well the system identifies discrepancies. 


Figure 6 
User Survey Response: Discrepancies 
How often does the system identify discrepancies in benefit amounts? 


Underpayments 


Overpayments 


m Never = Rarely = Sometimes = Most of the time = Always 


Source: Compiled by the Legislative Audit Division from survey responses. 


As shown in Figure 6, for both underpayments and overpayments, less than 35 percent 
of CHIMES-EA users think the system is identifying benefit discrepancies always or 
most of the time. A similar proportion believe the system is either rarely or never 
performing these functions effectively. 


Benefit Discrepancy Policy 


Policy clearly outlines rules for when overpayment and underpayment claims should 
be established and the rules that drive the procedures for how each situation should be 


handled. 


Procedures for supplemental payments are handled within the OPA office in a process 
that only requires supervisor approval if it is over $200. Overpayments include 
more controls and policy definition, such as supervisor approval for any claim to be 
established and an overpayment log to track supervisor reviews and approvals for 


central office review. 


In our review of these processes, we found that overpayment logs, while maintained 
at the OPAs, are not being consolidated for central office review. The department is 
trying to reduce the use of spreadsheets and increase efficiency by moving this process 
into the business intelligence tool, so the spreadsheets were not being consolidated. 
Without this consolidation and review of potential overpayments (including denied 
overpayments), the department does not have the ability to gather valuable information. 
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Gathering this information is another resource for performance monitoring as well as 
a control to oversee the overpayment process. Information gathered in overpayment 
logs includes elements of overpayment situations that identify where a system issue 
suggested an incorrect overpayment, or a user suggested an incorrect overpayment. 
Both of these situations are important to understand system functionality and improve 
user training. 


Adhering to policy and gathering the type of complete information captured on 
overpayment logs will assist the department in understanding where system controls 
can be strengthened in the benefit discrepancy process. The department will also 
be able to identify trends in overpayment causes that could imply a system issue or 
training issue. 
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RECOMMENDATION #5 


We recommend the Department of Public Health and Human Services 
conduct a review of consolidated, statewide overpayment information, 
including potential overpayments, on a periodic basis. 
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Manually Created Overpayments 


Our review of overpayments in the last two years, summarized in Table 5 below, 
indicates users are manually creating overpayment claims and denying most 
overpayment claims created by the system. Denied claims are claims that were created 
but not approved by supervisors. Pending claims are claims waiting for review by a 
supervisor. 


Table 5 
Comparison of Overpayment Claims in the Last Two Years 





Overpayment Status 





Source Denied Pending Approved 


System 2,392 2818 | 86% 370 18% 


User 464 14% 
Total 3,282 100% 












































Source: Compiled by the Legislative Audit Division. 





Overall, the system generated 60 percent of overpayment claims and users generated 
40 percent of the claims. System-generated claims totaled $1.8 million, whereas 
user-generated claims totaled $2.5 million, and on average are higher overpayments 
than system-identified overpayments. Table 5 (see page 24) shows that 87 percent of 
denied claims were established by the system, while only 18 percent of approved claims 
were established by the system. The majority of approved claims were established by 


users, which total $1.9 million. 


These metrics, plus our survey information and site observations, indicate there may 
be issues with system functionality related to generating accurate overpayment claims. 
However, we were unable to verify this due to lack of complete system documentation. 
Metrics may also imply that system-generated overpayments are possibly disregarded 


by users due to lack of system trust. 


Table 5 also shows that the majority of pending overpayment claims (yet to be approved) 
are system-generated. According to department policy the OPA has roughly three to six 
months, depending on when the claim is identified, to approve an overpayment. Of the 
2,818 pending claims established by the system, 2,263 ($575,000) are over six months 
old. We were unable to verify if all of the pending overpayments over six months old 
were valid or should have been denied. However, we did identify cases where staff 
noted the system-generated overpayment was incorrect, as well as cases that showed 
no reason for an overpayment to have been established. If these pending overpayments 
should have been denied, that increases the total number of overpayments created by 


the system and ultimately denied to over 5,000 in the last two years. 


The department was unaware of these metrics or any causes for the amount of manual 
overpayments. If the system is establishing overpayment claims inaccurately, extra time 
to verify this and create manual overpayment claims must be completed by the user. 
This increases the amount of time to work each case and increases mistrust of system 
functionality. System-generated overpayment claims are an intended function of the 
system to control overall accuracy. As such, industry standards require the department 


monitor this control to understand its effectiveness. Monitoring will help identify: 
¢ Why system generated overpayment claims are denied or possibly ignored. 
¢ Overpayment claims that were not approved in time. 


¢ Opportunities for system improvements. 
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RECOMMENDATION #6 





We recommend that the Department of Public Health and Human Services 
establish and document a process to review metrics related to the source of 
overpayment claims and address any identified system functionality issues. 


Es 


Overpayment Claim Management Process 


Once an overpayment claim is established, a process is followed until it is closed. 
This process involves two systems and divisions: CHIMES-EA in the Human and 
Community Services Division, and Accounts Receivable Management System 
(ARMS) in the Quality Assurance Division (QAD). Documentation of this process is 
depicted in the diagram below, with manual steps highlighted in red. 


Figure 7 
Claim Management Process 
Manual and System Interaction with CHIMES and ARMS 
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Source: Compiled by the Legislative Audit Division. 





Overpayments are established and approved by a supervisor in CHIMES-EA and 
then transferred to ARMS to manage the account. Once the overpayment claim is 


transferred to ARMS, CHIMES-EA only receives updated claim amounts. 


Changes to overpayments are sometimes necessary after more information is discovered 
or investigations are completed in the case of intentional program violations (IPV). 


The types of changes include adjusting the following metrics in a certain system: 
¢ Adjusting the claim amount (ARMS) 


¢ Adjusting the grant reduction amount if the client is still receiving benefits 


(CHIMES-EA) 
¢ Adjusting the penalty percent for TANF in the case of an IPV (CHIMES-EA) 


Manual Processes to Manage Overpayments 


Currently, only a few users have access to make these changes within the system. 
However, we noted manual work being done that increases the risk of inaccuracy in 
two instances: changing the overpayment occurrence time frame and creating manual 


overpayment claims. 


The process to change the overpayment occurrence time frame occurs when additional 
months need to be included in the claim, or the overpayment was established incorrectly 
as multiple claims for consecutive months instead of a single claim. In either case, the 
process requires the consolidation of multiple claims that includes manually creating 
and inactivating claims, manually transferring payments to new claims, and waiting 


for systems to update each other over a three-day span. 


Risk exists when multiple claims are established for consecutive months. Policy states 
instances like these should be established as one claim. ‘The system allows for claims 
to be set up at the month level because it is necessary, but this also makes it possible 
for an uninformed user to incorrectly set up an overpayment claim. There are reasons 
why it is important to set up the claim across all necessary months instead of for each 
individual month including: 


1. Each claim is a separate account in ARMS, so multiple claims means more 
accounts to manage in the system, and 


2. SNAP overpayment policy states that if a claim is less than $125 on a closed 
case, it is not recovered. For example, if multiple claims were set up for $100 
and not consolidated, this overpayment would not be recouped, when in 
fact, it should have been. 


Overpayments are reviewed by supervisors, but our review identified instances 
where these types of claims are not always identified and corrected. We reviewed a 
CHIMES-EA report of all overpayment claims, denied or approved by supervisors, in 
the last two years. The report was sorted to identify claims established in consecutive 


months. This review was not going to identify all consecutive claims, but did provide 
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some cases with incorrect claims established. The data showed that at least 91 claims, 
related to 24 cases, had overpayment claims set up incorrectly in this manner. Of these 
91 claims, 11 had been denied by the supervisor and recreated correctly, 6 claims were 
identified and consolidated after they were approved, and the remaining 74 claims 
were approved with the incorrect established time frame. None of these cases were 
overpayments established on closed cases, so we are unable to determine if overpayment 


claims were not recovered when they in fact should have been. 


By continually reviewing these processes, the department can increase awareness of 
how users interact with the system, as well as identify risks in manual processes and 
areas where controls need to be strengthened, such as user training for establishing 


overpayment claims. 


RECOMMENDATION #7 


We recommend the Department of Public Health and Human Services: 


A. Review claim management processes and controls to continually identify 
risks and opportunities for automated improvements to controls and 
efficiencies. 


Ensure all users are trained appropriately to set up and approve 
overpayment claims according to policy. 
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Chapter IV — Issue Management 


Introduction 


Issue management is the process for how system enhancements, defects, problems, 
errors, or anomalies are identified and addressed. The issues management process 
needs to be subject to well-defined procedures and close management supervision to 
ensure efficiency and reduce costs to the Department of Public Health and Human 
Services (department). The department uses a single system to manage this process. 
Issue management for the Combined Healthcare Information and Montana Eligibility 
System — Enterprise Architecture (CHIMES-EA) can be divided into three parts: 
identify and report the issue, analyze and prioritize the issue, and implement a solution 


to the issue, as shown in Figure 8. 


Figure 8 
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Source: Compiled by the Legislative Audit Division. 





Strong issue management controls include detailed, up-to-date documentation of 
procedures, oversight and involvement to ensure priorities are addressed appropriately, 
and monitoring of efficiency. Industry standards require these types of controls over 
issue Management to: 

1. Ensure standardized methods and procedures are utilized for efficient and 


prompt response, analysis, documentation, ongoing management, and 
reporting of issues. 
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2. Increase visibility and communication of issues to business and information 


technology (IT) support staff. 


3. Enhance business perception of IT through a professional approach in 
quickly resolving and communicating issues when they occur. 


4. Aligning issue management activities and priorities with those of the business. 


Maintaining user satisfaction with the quality of IT services. 


As part of our audit, we reviewed issue management controls the department has in 


place. Our audit reviewed these specific areas of control: 
1. Does the process align with industry standards? 


2. Is the process monitored for effectiveness? 


We obtained input from department staff to understand the process and reviewed 
how the process was managed within the issue management system. This assisted 
us in identifying differences between current documentation and what was actually 
happening. We then compared these findings to industry standards. From our review, 
we found that the process aligns with industry standards expect for how it is monitored 


for effectiveness and documented. Documentation is discussed in Chapter VI. 


Monitoring Issue Management for Effectiveness 


Our survey results showed that over half of the respondents felt issue management 
was effective some of the time, as show in Figure 9. Our audit work focused on 


understanding why there were not more users who responded favorably to this question. 


Figure 9 
User Survey: Issue Management 


How often is the issue management process effective? 


Sc 


mNever m@Rarely sSometimes sMostofthe time mAlways 


Source: Compiled by the Legislative Audit Division from survey responses. 





Monitoring the effectiveness of issue management ensures the process is meeting the 
needs of users, clients, and the department. It also allows the department to identify 
areas of improvement to strengthen the relationship with users. We reviewed three 


aspects of effectiveness: 


¢ Timeliness: Timely resolution of issues is not only more cost effective, but 
ensures disruptions to productivity are minimized and the use of temporary 
workarounds is limited. 


¢ Root Cause Analysis: Root cause analysis is important so the issue does not 
persist and potentially cause multiple iterations within the process. ‘This saves 
the department time and money, as well as reduces user frustration with 
encountering an issue multiple times. 


¢ Communication: Communication is important so users understand time 
frames for temporary workarounds and to ensure users are aware of actions 
taken and plans to prevent future incidents from occurring. ‘This also instills 
trust with users throughout the state who may not have much direct contact 
with system managers. 


Timeliness of Issue Management 


When asked if users felt issue management was timely, responses were split between 


favorable and unfavorable, while the majority responded Sometimes. 


Figure 10 
User Survey Response: Timely Resolution 


How often are issues being addressed in a timely manner? 


m= Never = Rarely = Sometimes = Most of the time m= Always 


Source: Compiled by the Legislative Audit Division from survey responses. 





We ran reports within the issue management system for the age of defects, or system 
issues, that were identified but not yet resolved to understand how it coincided with 


survey responses. This report indicates the average age of defects yet to be resolved is 


160 days. 


When asked about this, the department noted that the number may be misleading 
because there are multiple levels of priorities within the group of open defects. Issues 
that are lower priority and do not need to be addressed immediately are offsetting the 
issues that are being addressed sooner. According to the department, there is also a 
backlog of defects and enhancements that need to be cleaned up. ‘The average age of 
defects could be thrown off by duplicates or issues that have already been resolved but 


are not marked so. 
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Prioritization Within Issue Management 


To understand the age of defects, we reviewed the prioritization process and how 
effective it is at increasing the efficiency of issue management. There are two stages in 
which priority is determined for issues within the system, as shown below in Figure 11. 
The nature and details of the issue determine the priority of the help desk ticket. The 
outcome of the Business Value Assessment (BVA) determines the priority of the defect 
or enhancement. 


Figure 11 
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Source: Compiled by the Legislative Audit Division. 





Help Desk Ticket: During our fieldwork, the level of prioritization utilized for 
help desk tickets was critical issuance. Critical issuance tickets need to be addressed 
within 48 hours because the process for issuing benefits is halted. If an issue was not 
categorized as critical issuance, it was left blank. During fieldwork, the department 


identified this as an issue and was starting to revise this process. 


Defect or Enhancement Ticket: These tickets are given priority levels that include 
critical, high, medium, low, and no priority. The BVAs for defects and enhancements 
are used to determine these priorities, but reports show that only 23 percent of 
enhancements and defects have been assessed. At the time of our review, 466 BVAs 


were completed and 1,526 were incomplete. 


These areas were identified as bottlenecks in the process: analyzing root cause of the 
help desk ticket, and completing the BVA for the enhancement and defect. The BVA 
consists of 15 questions, ranging from federal fiscal impact to public perception and 
number of clients impacted. This assessment is thorough and includes discussion 
between the contractor and the department to finalize, which takes time. Issues 
outnumber the amount of staff available to spend time analyzing them thoroughly, 


and without priorities, this work is not focused to areas of need. 


Both of these processes are necessary for issue management. However, without 
appropriate priority levels and the completeness of prioritization, the benefit of 
prioritization is lost and efficiency of the process is hindered. Monitoring the age 
of help desk tickets with only two categories for priority and monitoring the age of 
defects when they have not been prioritized does not provide useful information for 
the department. ‘The limited priorities and unprioritized defects also do not provide a 
helpful direction for what tickets to address first. The department recognized this issue 


as well and is starting to implement more priority levels for help desk tickets. 


Industry standards state all enhancements and issues be properly prioritized. 
Prioritization allows for better planning and strategy to complete work more efficiently. 
By ensuring priorities are assigned and accurate, the department will be able to monitor 


the age of tickets and timeliness of the process based on each priority level. 


RECOMMENDATION #8 


We recommend the Department of Public Health and Human Services 
ensure all current help desk tickets, enhancements, and defects have been 
prioritized. 


0 ETT 


Departmental Governance of CHIMES-EA 


The department does not monitor timeliness of issue management. Monitoring is vital 
to ensuring the process is efficient and timely, especially when the issue management 
process is a coordinated effort between the department and a contractor. Through our 


review, we found the department relies on the contractor for these key processes: 
¢ Manage documentation 
¢ Help complete BVAs to prioritize defects and enhancements 


¢ Set the baseline for what issues are addressed in each system update 
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The current process has points of contact between the department and the contractor 
at Change Control Board meetings and quarterly status meetings when the contractor 
reports progress and expected dates for department testing of system updates. ‘They 


also communicate individual issues throughout the reporting and classification stage. 


Industry standards state that if business need requires a government agency to rely 
on a contractor, an effective compliance monitoring process must be established by 
the agency. It may be necessary for the contractor to assist, or even manage, certain 
processes. If this is the case, then strong oversight within the process, along with 
establishing clear performance benchmarks and standards for the contractor, must occur 
for maintaining and improving the system. Without these controls, the contractor may 
not act in the best interest of the department. In the case of CHIMES-EA, impacts 


can occur during critical points of the process: 
1. Determining whether an issue is an enhancement or defect. 


2. Determining priority and timing of the issue being fixed. 


The agreement between the department and contractor assists in governance of the 
process. It outlines performance standards for timeliness of issue management based 
on the severity of the issue. This document also outlines monetary penalties if those 
standards are not met. ‘The performance standards require the contractor create a 
plan of resolution within a certain time frame for each severity level. However, it does 
not address a time frame for implementing a solution for the issue. A recommended 


resolution and a plan to correct the issue are all that is required. 


When reviewing the current process, we noted that it does not classify the severity level 
of an issue until it is determined to be a defect or enhancement, as shown in Figure 12 
(see page 35). ‘This indicates an issue has to go through the help desk ticket process 


before it is given a severity and can be monitored against the performance standard. 


Figure 12 
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Source: Compiled by the Legislative Audit Division. 





Our review also noted the department has not been tracking timeliness in regards to 
severity and the performance standards. The department states it has not had issues 
with these standards or had to enforce them, but was unable to verify this through 
documentation. Our review of issue management system reports shows the age of 
unresolved, high severity defects is 200-250 days. With standards that require the 
contractor to provide a plan and not a resolution within an established time frame, 
contractor performance related to efficiency cannot be enforced. Without this pressure, 
the average age of unresolved defects and efficiency of the issue management process 


may not improve. 


To govern the process and ensure it is efficient, industry standards require an agency 
set benchmarks for the time frame of solutions and monitor against those benchmarks. 
By establishing performance benchmarks for the contractor and monitoring the 
contractors performance in critical areas, the department can confirm the contractor 
is meeting business needs for assigned responsibilities. This will allow the department 


to govern the process and enforce penalties to ensure the process is working efficiently. 
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RECOMMENDATION #9 





We recommend the Department of Public Health and Human Services 
establish, monitor, and enforce benchmarks for: 


A. Analysis and prioritization of help desk tickets, defects, and 
enhancements. 


B. Time frames for implementing solutions. 


_ ‘i 


Root Cause and Classification of Issues 


We reviewed help desk tickets and defects, ran reports within the issue management 
system, and discussed root cause analysis with department staff. Root cause analysis 
helps to classify an issue initially: training issue, system issue, enhancement, etc. 
Classification helps understand what the next steps are in solving it. These classifications 
are also used in dashboards within the issue management system for the department to 


identify training issues as compared to system issues and enhancements. 


This process requires the help desk ticket to be linked to a defect if the system is the 
root cause of the issue. In instances of training issues or policy questions, a defect 
would not exist to be linked to the ticket. The number of help desk tickets linked to 
a defect ticket is important when completing the Business Value Assessment (BVA) 
and prioritizing the defect. Our review of help desk tickets through site observations 
and our own testing showed that tickets were not always linked to defects or classified 
correctly. Examples of inconsistencies found include: 


¢ Help desk tickets documenting the error as known, but not referring to the 
defect intended to address it. 


¢ Help desk tickets identified as a training issue when a defect is linked to the 
issue because the root cause is due to system error. 


Industry standards require all issues be properly classified. Incorrect and incomplete 
classification of issues and not linking issue tickets to defects disrupts the prioritization 
process and provides the department with inaccurate data for reporting and viewing 
issue cause. The department acknowledges the need for a more consistent process 
that is understood by everyone and has been making efforts to be more accurate and 


consistent when classifying issue tickets. 
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RECOMMENDATION #10 


We recommend the Department of Public Health and Human Services: 


A. Improve the process by ensuring all classifications on help desk tickets 
are accurate and complete and defects are associated as necessary. 


B. Document the process to ensure consistency. 


es 


Addressing Root Cause 


For system issues, further classification is completed to understand how the root cause 
will be addressed and whether the defect is data or application related. 


¢ Data Defect: Addresses the instance of data that is incorrect; no code 
changes are made to adjust how the system functions. 


¢ Application Defect: Addresses how the system functions by adding/ 


changing code. 


This classification was reviewed through issue management system reports. We 
evaluated open and closed defects by this classification and noted differences between 


the amount of data defects versus application defects since implementation. 


Table 6 
Defects Identified in Production Environment 





CLOSED OPEN 





Type # of Defects % of Total # of Defects % of Total 














Data Defect 133 13% 











Application Defects 3% 





Other Defects 1% 
Total 2,804 100% 


























Source: Compiled by the Legislative Audit Division. 





The high number of data defects that have been resolved in comparison to application 
defects leads us to believe that the root cause, the reason the data is incorrect, is not 
being addressed. Reviewing these reports over a 30-day timespan also showed that 
closed data defects increased by 526, while closed application defects only increased 
by 16. We did not complete testing due to the amount of defects, but we identified an 
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instance of three data defects with the same cause that had been fixed with no obvious 


application fix. 


There are situations when a data defect is addressed as a temporary patch until the 
root cause can be addressed through an application defect. There are almost three 
data defects to each application defect, including both open and closed defects. ‘This 
indicates that data defects are being done as patches multiple times before an issue is 


addressed through an application fix. 


We cannot assure that all issues are being addressed due to inconsistent classification 
and linking of issues, but have evidence that not all issues are being addressed at 
the root cause of system functionality. The department states data defects address 
individual issues and are used as interim solutions to issues. This is a necessary process 
within industry standards; however, standards also require implementing a sustainable, 
long-term fix to reduce the risk of business disruptions, reoccurrence of issues, and a 
user perception that issues are not being fixed within the system. Without controls in 
place to ensure the root cause is addressed, data defects become a quick solution and 


the underlying cause is not addressed. 
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RECOMMENDATION #11 


We recommend the Department of Public Health and Human Services 
establish a process to limit reliance on data fixes and ensure data fixes are 
associated with a long-term application fix, if necessary. 


Es 


Communication of Issue Management 


Our survey also asked a few questions about communication in regards to issue 
management. We asked specifically about issues the user had identified and general 


communication about all issues within the system. 


Table 7 
User Survey Response: Issue Communication to Users 





Communication users have received concerning 
issues from either management, central office, or Responses* 
help desk/service desk. 





Issues being fixed with upcoming build 67% 





Known Issues 50% 


None 13% 








Other (please specify) 18% 








Communication users have received concerning 


issues they have reported. Responses 





Issue or Ticket number 80% 





Request/Follow-up for Issue details 39% 





Update on issue status 33% 





Notification of when the issue will be fixed 33% 


Reason for closing the issue (fixed or not) 





| don’t report issues 





Other (please specify) 











Source: Compiled by the Legislative Audit Division from survey responses. 


*Respondents could identify multiple options. 





From these results, communication appears to be occurring in some instances more 
than others. For issues the respondent identified, most receive acknowledgement with 
an issue or ticket number. However, the number of respondents that receive other 
types of communication drops sharply. When discussing issues as a whole, about half 


of the users responded that they received communication. 


We reviewed issue management system reports and communication sent from 
the central office to understand why there was not a higher percent of respondents 
receiving communication. A metric within the issue management system tracks 
whether or not the user was contacted about resolution. When looking at all closed 
tickets, over 22,000 had the answer “No,” and over 29,000 had the answer “Yes.” This 
indicates that 44 percent of ticket resolutions have not been communicated to the 
user. Through site observations and contact with users, we verified tickets were closed 
without communication. As an example, when we contacted a user about a closed 
ticket, the user contacted the service desk to ask why the ticket was closed. ‘The issue 


was subsequently reopened. 


CO GEE 


39 


40 | Montana Legislative Audit Division ‘ Legislative Audit Division 


We also examined communication sent to users about planned system updates and 
reviewed training information about system changes with each update. We found 
training for system improvements that users are required to take; however, this training 
does not review all changes within each update. Only a brief description of what 
was changing within each update was noted in spreadsheets shared with users. ‘This 
spreadsheet does not include detailed information for a user to understand what exactly 
the enhancement or issue was or information on whether or not their interaction with 


the system needs to be modified due to the change. 


Department staff agree that communication needs to be increased, but said they do 
not communicate some things, like temporary actions addressing issues, so users do 
not misapply them to situations when the action is not necessary. They also want to 
know how often issues occur, and they feel they would lose sight of this if users were to 
know all of the temporary actions used. The department stated that due to the number 
of temporary actions, managing a list like this would be a daunting task as well, so 


these actions are handled on a case by case basis and no master list is maintained. 


Industry standards note that communication is necessary to keep users informed of 
time frames for temporary actions and to ensure that the users affected are aware of 
the actions taken and the plans developed to prevent future incidents from occurring. 
Communication is important to decrease the risk of users applying temporary 
actions when no longer necessary or using them when not approved. By increasing 
communication and transparency to users about issue management, user trust will be 


increased and risk of system misuse and decreased efficiency will be mitigated. 
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RECOMMENDATION #12 


We recommend the Department of Public Health and Human Services 
increase communication to users regarding: 


A. — Individual help desk ticket resolution and any related defect resolutions. 


B. Defects being addressed through each build and what workarounds will 
no longer be necessary. 
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Chapter V — Access and Monitoring 


Introduction 
Access and monitoring controls ensure the integrity of the data within the system and 
the integrity of the programs managed by the system. Controls include: 

¢ Process for granting access that ensures access is needed and approved. 

¢ Process for assigning access that: 


) Ensures only the needed functions are assigned and that users are not 
given privileges for when they do not have a business need. 


) Ensures segregation of duties for processes with a high risk of fraud. 


¢ Process for reviewing all access on a consistent basis to identify changes and 
updates. 


¢ Process for monitoring privileged functions to ensure they are not abused 
and are accurate if overriding system suggestions. 


Weaknesses in these processes increase the risk of unauthorized actions, fraud, and 
data errors that ultimately affect the accuracy of the benefits issued and integrity of the 
programs and department. They also increase the risk of personal identification and 


health information security issues within the system. 


We reviewed documentation and user lists, and discussed processes with department 


staff to understand what processes were in place to mitigate these risks. 


Privileged Function Controls 


Privileged functions include any functions that are not given to the general population 
of users. Privileged function control helps to mitigate the risk of fraud, security issues, 


or data integrity issues. Within CHIMES-EA, these functions specifically relate to: 

¢ ‘The ability to override or change system metrics or outcomes by: 
Issuing manual benefits 
0 Creating manual overpayment claims/supplemental payments 
Q Issuing manual correspondence and mass mailing correspondence 
Changing redetermination dates 

¢ Managing user access by creating, updating, or deleting user roles 

¢ = Specific functions that only certain individuals are authorized to do: 
0 Work Readiness Component (WoRC) functions 
0 Child Support Enforcement Division (CSED) custodial parent 


functions 


Q Overpayment adjustment functions 
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We reviewed access by obtaining a user list and definitions of privileged roles to 
understand what functions individuals were given access to. We identified access that 
was not necessary including: 

¢ Four roles, assigned to 83 users, with access to adjust overpayments. 

¢ ‘Two roles, assigned to 12 users, with access to send mass correspondence. 


¢ Four users with roles assigned that were no longer necessary. 


These issues exist because the current access review procedure does not include a 
periodic review of what privileged functions are allowed for each role and if the function 


is necessary, or a review of what users have roles with access to privilege functions. 


Industry standards indicate the access process not only include a review of all users, 
but a periodic, thorough review of privileged roles to ensure functions allowed to each 
role still align with business processes and needs, as well as a review of the users who 
have those roles. By doing this, the department will decrease the risk of users having 


more privileges than necessary. 
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RECOMMENDATION #13 


We recommend the Department of Public Health and Human Services 
improve current access review procedures by including: 


A. What privileged functions are allowed in each role and if it is necessary. 


B. Which users have roles with access to privileged functions. 
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Manual Change Controls 


Manual changes to the system include both user and contractor processes to adjust or 


override information within the system. Our audit reviewed both types of changes: 
¢ Manual changes to system determinations and calculations by users. 


¢ Manual changes to data within the system by the contractor. 


In the case of a system issue, manual changes may be necessary. However, proper 
controls need to be implemented to ensure data integrity. Industry standards stress 
the importance of segregating responsibilities in processes like these. In cases where 
separation cannot be accomplished, monitoring of user activity within these processes 
should be implemented. Strong controls over these processes are important to mitigate 


the risk of fraud and uphold the integrity of the program. 


Manual Changes by the Contractor 


Within our review of the issue management process, we identified a process to adjust 
data within the system when issues occur. These adjustments can be done to various 
data elements of a case including important dates and eligibility status. These changes 
can be done to individual cases, or mass data changes are done if multiple cases have 
the same problem. ‘This process includes proper steps to review the proposed change, 
including the contractor directly accessing the system database to make the change 


after these approvals. 


When discussing contractor access with the department, it was unclear what type 
of access the contractors had to the system database because this is not reported or 
managed. The only documentation of contractor access is the initial access request. We 
reviewed the requests of the contractor's staff tasked with administering these changes. 


These requests showed full administrative access to the system database. 


Industry standards stress that contractors should not have access to live data within 
the system database; however, when that cannot be achieved, proper monitoring needs 
to be established. If this level of access is granted to the contractor, industry standards 
and state policy require management ensure only authorized changes are being made. 
Without management, individuals outside of the state agency can access personal 
information and adjust important case information without proper authorization or an 


audit trail for detection. 


By managing the access of contractors within the databases, the department reduces 
the risk of system inaccuracies and data integrity issues. Managing access could include 


the following activities: 
¢ Maintaining and continually reviewing documentation of system access. 
¢ Monitoring the activity of users with privileged functions. 
¢ Reviewing logs of changes made to the database. 
¢ Establishing a process to identify unauthorized access and changes. 


¢ — Limit access to only times when its needed. 


a 


RECOMMENDATION #14 


We recommend the Department of Public Health and Human Services: 
A. Document and track contractor access to system databases. 


B. Monitor contractor activity within system databases. 


SCS 
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Manual Changes by Users 


Within our review of user access, we also evaluated processes with a high risk of fraud 
to ensure proper segregation of duties exists. The processes that increase these risks 
are those that give users the ability to manually change system determinations and 
calculations. The processes identified include: 


¢ Manual Issuance: Issuing benefits prior to the system determining eligibility 
and calculating benefits. 


¢ Overrides: Changing system calculated benefit amounts or eligibility 
determinations. 


¢ Discrepancies: Manually establishing a supplemental payment or 
overpayment claim. 


Survey responses show that users are more likely to make changes to the system than 
not (shown in Figure 13), which increases the importance of strong controls. 


Figure 13 
User Survey Response: Manual Changes 


How often are you making manual changes? 


= Always = Most of the time = Sometimes = Never 


Source: Compiled by the Legislative Audit Division from survey responses. 





When reviewing controls for high risk processes, we found that 429 users can manually 
create overpayment claims, but the system requires dual authorization and access to 
adjust overpayment claims is limited. While this is an adequate control, the following 
two issues were identified in other high risk processes: 


1. Manual issuance and manual supplement capabilities are limited to only 
supervisors and a few central office staff (85 users) with no approval process. 


2. Override capabilities are assigned to most users (over 450) with no approval 
controls within the system; however, the department insists users not 
complete an override until consulting with central office first. 


While there are some controls in place for these processes, the lack of approval controls 
increases the need for monitoring of manual issuances and overrides. When discussing 


how these are monitored, the department stated they are not directly monitored. 


They have run a report in the past to show the amount of overrides compared to 
authorizations, but do not consistently run it or use any other reports to ensure overrides 
are accurate and being used appropriately. That report showed that overrides steadily 
increased from December 2013 (3 percent of authorizations) to June 2014 (7 percent 
of authorizations). In 2015, that number has maintained between 3 and 6 percent of 


authorizations for each month. 


The department is currently relying on users reporting issues as a means of 
understanding how often manual changes occur. This method does not allow for easy 
understanding of all overrides or manual issuances, and is only accurate if users are 
reporting all issues. Survey responses showed that not all users are reporting manual 
change reasons. 


Figure 14 
User Survey Response: Issue Reporting 


How often do you report the reason for manual change to service desk? 


m Never m Rarely = Sometimes = Most of the time m Always 


Source: Compiled by the Legislative Audit Division from survey responses. 





By monitoring manual issuance and overrides, the department can ensure the changes 
to the system are accurate. They will also be able to understand how often changes are 
happening instead of relying on users to report when manual changes happen. ‘This 
control activity will reduce the risk of fraud and ensure data and program integrity. 


a 
RECOMMENDATION #15 


We recommend the Department of Public Health and Human Services 
strengthen manual change controls by implementing procedures to monitor 
manual issuances, manual supplemental payments, and overrides. 
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Chapter VI — Defining System Functionality 


Introduction 


Defining system functionality is important to understanding how the system should 
function and knowing when it is not functioning properly. This is done through 
proper training and thorough documentation. Industry standards require these both 
be adequate to increase the understanding of how the system should function or how 
the process to manage the system works. Other reasons thorough documentation and 


training are necessary include: 
¢ Reducing system errors that go unidentified. 
¢ Reducing improper system management and use. 
¢ Improving inefficient processes. 


¢ Ensuring all necessary steps are documented. 


Our audit reviewed controls in both training and documentation: 


¢ System Training: We referred to user guides and system training courses 
provided by the Department of Public Health and Human Services 
(department) to understand how to use the system. We also talked about 
these resources with users. 


¢ Documentation: We reviewed documentation describing system design, 
expected functionality, and processes involved in managing the system. 


Our audit noted documentation that was incomplete, out-of-date, or that did not exist, 
as well as training that needed to be strengthened. The following sections discuss our 


findings and recommendation to the department. 


System Training 


Through site observations, users expressed concerns with the amount of training a 
person receives before having to work actual cases. They indicated that because it 
did not cover important points of policy and processes, issues were occurring with 
case management. We observed users relying on experienced staff with a good policy 
background to direct them when they have issues because that is the only resource 
available. As experienced staff leave, knowledge goes with them. ‘This over reliance on 
other staff creates inefficiencies by taking time away from both the person needing 
guidance and the person providing guidance. If a user is not aware of how the system 
is supposed to enforce policy, or act in certain situations, the risk that the user may not 


identify when the system is not functioning correctly is also increased. 


rr, SPOT 


47 


48 | Montana Legislative Audit Division ‘ Legislative Audit Division 


User guides and system manuals are considered system documentation that describes 
the interaction between users and the system and defines how the system should 
function. Industry standards require these types of documents, as well as training, 
be reviewed and updated on a regular basis to ensure adequacy. Guidelines suggest 
reviewing user satisfaction levels for training and operation manuals after training is 
conducted and after the user has interacted with the system to better understand how 
well users were prepared. The department currently has a survey available for users 
immediately after training. However, this does not gather information to understand 


how well the user was prepared for working in actual cases. 


The department is initiating plans to review this type of material and make adjustments 
to training with estimated completion in 2016. It is important this initiative be 
completed as soon as possible to improve the user experience with the system, improve 
efficiency, and reduce the risk of benefit errors. Reviews of training material and 
manuals should be conducted on a regular basis after reviewing user satisfaction to 


continuously improve the training process. 


Re 
RECOMMENDATION #16 


We recommend the Department of Public Health and Human Services 
improve user training by: 


A. Covering scenarios that introduce more policy. 


B. Evaluating how satisfied users are with the amount of preparation and 
training they received. 


© 


Update training on a regular basis. 


BS 


System Documentation 


System documentation creates the baseline for how the system is intended to function. 
This documentation also outlines the processes the system is involved with and how it 
is intended to be used. Our review of the various types of system documentation are 


summarized below: 


Business Rules: Identify what actions are taken when certain conditions exist. These 
documents are the blueprints for how the system processes situations and makes 
decisions. Each rule is outlined in one document, so there are over 900 documents that 
have to be maintained to understand how the system should function. Our review of 


253 business rule documents noted: 


¢ For 33 business rule documents, we were unable to find a condition or action 
noted in policy within the document, the wording was unclear or confusing, 
or it was not up-to-date. 


¢ For 26 business rule documents, there was no reference to policy, or an 
incorrect reference was noted. 


Correspondence: Communication to clients concerning their case or application 
is considered correspondence. Design documentation states how correspondence is 
triggered, either by the user or by the system, and what conditions need to exist if 
the correspondence is triggered by the system. These design documents also show 
the layout and standard language of the correspondence. Our review of 152 design 
documents for correspondence noted: 


¢ Nine correspondence documents missing details for how the correspondence 
is triggered or conditions for it to be triggered. 


¢ Twenty-nine correspondence documents that were not referenced in policy. 


Interfaces: Interface design documentation provides expectations for how an interface 
is intended to function and be monitored. Interface documentation should also include 
an agreement between both system managers for security purposes. These agreements, 
known as Data Sharing Agreements (DSAs), ensure that the data will not be misused 
and describe security requirements and the nature of the information communicated. 
Our review of 39 interface documents noted: 


¢ ‘Two interfaces did not have data sharing agreements. The department is 
waiting for the other system managers to finalize the documents. 


¢ Sixteen interfaces where documentation was inconsistent with either how 
often the interface was run, or the information monitored and what was 
documented to be monitored. 


Benefit Discrepancies: ‘These documents include business rules defining how a 
discrepancy is determined by the system and separate documentation defining the 
process to manage discrepancies within the system. Our review of the process and 
associated system documentation noted that the following types of documentation do 
not exist: 


¢ Documentation of the process to manage discrepancies, noting all user 
interaction with the system and information transfers. 


¢ Documentation determining how the system calculates overpayment claims. 


Issue Management: Issue management documentation is critical to ensure 
standardized procedures are being followed and that these procedures reduce risk 
and inefhiciencies. From the audit work performed, we determined there is an issue 


management process established, and the process expectations set by the department 
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are followed by the contractor. However, we found outdated and inaccurate plans and 


system documents that had been updated but lacked department approval. 


System Testing: Documentation throughout system testing is important to ensure 
testing is thorough and approved before changes are made to the system. This reduces 
the risk of issues occurring due to the change. We reviewed test scenarios specific to 
correspondence within the system and the testing process within issue management. 
Test scenarios identify the exact aspect of the system that is being tested by identifying 
steps that should be taken to test and what the expected result should be. Our findings 
include: 

¢ Forty-eight notices were not tested consistently with design documentation 


defining the triggers (automatic or manual) to create the correspondence. The 
department stated that thorough, yet undocumented, testing was conducted. 


¢ ‘There is no documented process for testing that includes: 
) | What test phases are appropriate and expected to be used. 
0 Criteria for evaluating success to ensure it meets business expectations. 
0 Roles, responsibilities, and authority for approving testing. 


¢ ‘Test scenarios are being created for testing; however, they are not always 
executed. User testing only covered 31.8 percent of the test scenarios created. 
The department noted that other testing was done and that for smaller 
updates, testing is noted in the issue management system instead of the 
normal testing system since it is quicker and easier. 


These incomplete documents are the result of a lack of department oversight to ensure 
documentation is complete and accurate with changes made to processes and the 
system. While the department has assigned responsibility of updating and maintaining 
documentation to the contractor, the department is accountable for ensuring accuracy 
and completeness. Without this documentation, we cannot conclude what certain 
intended system functions are or that processes to manage the system are consistently 


occurring according to policy or industry standards. 


Policy and Procedural Documentation 


Policy and procedural documentation outline how interactions are expected to occur 
with the system. They also provide users with an authorized set of procedures and 
policies driving specific processes relative to the new system. This decreases the risk that 
the system is misused and helps users to identify when the system is not functioning 


according to policy. Our review of this documentation is summarized below: 


Policy: Policy currently references the previous system and procedures within that 


system. Because of this, new users have to rely on experienced staff as the resource for 


current operations. If the policies or procedures are not accurate and do not reference 
current business systems and operations, employees have no relevant documented 


procedures to follow. 


User Guides: When testing the system, we were unable to find training or procedural 
documents for some of our scenarios. [hese scenarios were not documented within user 
guides well enough to inform us how to correctly enter information. Some examples of 
scenarios we noted include: 

¢ = Self-employment 

¢ Alien sponsor/legal alien 

¢ Reviewing a supplement/overpayment claim suggested by the system 

¢ Establishing a manual overpayment claim according to policy 


¢ Individuals on strike 


We also witnessed users not knowing what to do in the same scenarios. Users had 
nothing to refer to because training documents or user guides do not cover the subject, 
and policy references the old system. These users had to set the case aside to discuss 


later with a more knowledgeable user. 


Managing Documentation 


The department relies on the contractor for managing most of the system 
documentation, but is accountable for ensuring it is accurate and up-to-date. This 
is why change logs are a part of each system document and require the change be 
documented by the contractor and approved by the department. Our audit found the 
department is not consistently reviewing/auditing documentation to ensure updates 


are accurate and completed, or to ensure all system decisions are documented. 


Policy and procedural documentation updates were delayed until business processes 
were finalized and the work could be done in one review. The department is initiating 


that review now and plans to have the work completed in 2016. 


By increasing controls to maintain more accurate and complete documentation, the 


department increases awareness and understanding in the following areas: 
¢ How the system should be functioning. 
¢ How to properly use the system. 


¢ How to manage processes that control the system. 


51 


52 | Montana Legislative Audit Division ‘ Legislative Audit Division 





es 


RECOMMENDATION #17 


We recommend the Department of Public Health and Human Services 
create or update documentation and strengthen controls for reviewing 
documentation in the following areas: 


A. 


oO mm O O W 


Policy referencing previous systems 

Business Rules and Correspondence Detailed Design Documents 
Interface Design Documentation and Data Sharing Agreements 
Benefit Discrepancy Business Rules and Process 

Issue Management Procedures 

System Testing Plans 


User Guides and System Manuals 


SE 


Appendix A 


‘This section provides survey response data. Throughout the survey users were also given opportunities to 
comment on certain topics. Those comments were not included in this summary. The total population 
for the survey was 984, with 512 responses and 472 nonresponses. Throughout the survey, skip logic was 


used so users were directed to questions relevant to their position and interaction with the system. 


Survey Data Used for Report 

What is your role within the system? 

Total Responses: Nonresponse: 0 
Eligibility Levels 1-5 
Eligibility Supervisor 
WoRC Related 
CSED Related 
Regional Manager 
Program Officer 
Director 
Accounting Services 
Investigator/Auditor 
Legal Services 
Not sure 
Other (please specify) 
What do you spend most of your time in CHIMES EA on? 


Total Responses: Nonresponse: 0 


Case Management (intake, authorization, update) 

Case Assistance/Troubleshooting 

Supervisory Actions (approvals, overrides, reviews) 

Read/View Information Only 

Other (please specify) 

How long have you been in your current position? 
Total Responses: 512 Nonresponse: 0 

0-2 years 182 

3-5 years 108 

5-10 years 99 

10+ years 123 


How long have you been working with Human and Community Services? 


Total Responses: 512 Nonresponse: 0 
0-2 years 99 
3-5 years 91 
5-10 years 84 
10+ years 
| don't work in this Division 54 





On average, how many hours do you spend working directly in CHIMES per day? 
Total Responses: 512 Nonresponse: 0 
More than 5 hours 310 


53 


3 hours to 5 hours 62 
1 hour to 3 hours 61 
Less than 1 hour 19 


If you were involved in the development of CHIMES EA, please select your role(s). 


Total Responses: 512 Nonresponse: 0 

None, was not involved 431 
Consulted on Policy during development 23 
Testing the system prior to implementation 62 

29 
In your opinion, is CHIMES EA... 

Most of the 
Total Responses: 485, Nonresponse: 27 time Sometimes Never 

User Friendly 138 212 69 
Responding in a timely manner 163 212 36 
Easy to learn 159 216 92 
Processing applications quickly 134 248 58 
Accurate 146 291 2/ 
Simplifying Processes 117 231 
Reducing Errors 88 256 


Allowing you to spend the majority of your time 
on qualitative customer care 97 239 


Increasing the efficiency of collection, 
reporting, and analysis of client data 


What functions does CHIMES provide that assist you in your daily work? 





Total Responses: 485 Nonresponse: 27 
Information gathering in one system 275 
Eligibility determination 291 
Benefit calculation and issuance 234 
On-going case management 267 
Staff workload management 126 
Reporting 114 
System Interfaces 201 
Not Sure 30 
Other (please specify) 62 
Do you feel these functions are working as intended? 
Total Responses: 485 Nonresponse: 27 
Yes 117 
No 260 
Not Sure 108 
What best describes your role in CHIMES? 
Total Responses: 481 Nonresponse: 31 
| enter information in to the system 284 


| supervise/manage people who enter 
information in to the system 59 
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| assist people who enter information in to the 


system, but do not enter information 6 
| view information, but do not enter information 93 
Other (please specify) 39 


Does your office have a policy/procedure for addressing discrepancies in benefit amounts that have been 
issued? 


Total Responses: 339 Nonresponse: 4 

Yes 228 

No 27 

Don't Know 40 

Not Applicable 44 

Does the system assist in identifying discrepancies in benefit amounts issued? 
Total Responses: 339 Nonresponse: 4 

Yes 137 

No 140 

Not Applicable 62 


In your opinion, how often does the system identify discrepancies in benefit amounts? 


Overpayments Underpayments 
Total Responses/Nonresponse: 229/114 227/116 


Always 1 

Most of the time 52 
Sometimes 84 
Rarely 3/ 
Never 13 
Not Applicable 36 


What type of manual changes do you make within CHIMES to get a desired result? 


Total Responses: 333 Nonresponse: 10 
Overriding benefit amount 196 
Open/Close of a case 179 
Withdraw of an application 136 
Other (please specify) 74 
Not Applicable 45 
None 21 
In your estimation how often are you making manual changes? 
Total Responses: 333 Nonresponse: 10 
Always 15 
Most of the time 66 
Sometimes 161 
Rarely 29 
Never 13 
Not Applicable 49 





When a manual change is done, how often are you notifying Service Desk of the reason for the manual 
change? 


Total Responses: 333 Nonresponse: 10 


Always 47 


Most of the time 82 
Sometimes 73 
Rarely 51 
Never 23 
Not Applicable o7 


In your estimation, how often do you create manual budgets to verify CHIMES budgets? 
Total Responses: 333 Nonresponse: 10 

Always 11 

Most of the time 43 

Sometimes 117 

Rarely 78 

Never 17 

Not Applicable 67 


How often do the cases that you work require some kind of workaround due to a system issue? 


Total Responses: 332 Nonresponse: 9 
Always 
Most of the time 
Sometimes 
Rarely 
Never 
Not Applicable 
Are you aware of an available list of authorized workarounds for users to reference? 
Total Responses: 332 Nonresponse: 9 
Yes 60 
No 249 
Not Applicable 23 
How many workarounds currently exist that you are aware of? 
Total Responses: 332 Nonresponse: 9 
Less than 5 76 
5-10 69 
10-20 64 
20+ Do 
Not Applicable 68 
Are you required to report issues found in CHIMES? 
Total Responses: 466 Nonresponse: 46 
Yes 303 
No 163 
Does your office have a process for reporting issues within the system when they are found? 
Total Responses: 306 Nonresponse: 0 
Yes 286 
No 7 
Don't Know 13 





How do you report issues? 


Total Responses: 306 Nonresponse: 0 
Through my supervisor/manager 218 
Direct call to a Service Desk 94 
Email to a Service Desk 98 
| don't report issues 6 
Other (please specif 45 


In your estimation, how often do you report problems in the system that you encounter even if you use a 
workaround to move on? 


Total Responses: 306 Nonresponse: 0 
Always 
Most of the time 
Sometimes 


Rarely 


Never 
Not Applicable 
Please select what communication you have received concerning issues from either management, central 
office, or help desk/service desk. 
Total Responses: 305 Nonresponse: 1 
Known Issues 151 
Issues being fixed with upcoming build 205 
None 40 
Other (please specify) 56 
Are you aware of authorized temporary actions to work around issues until fixed? 


Total Responses: 305 Nonresponse: 1 


Yes 80 
No 88 
| Know how to work around the issues, but not sure if the 

actions are authorized 100 
Not Applicable 37 


In your estimation, how often are issues that affect your daily work being addressed in a timely manner? 
Total Responses: 305 Nonresponse: 1 

Always 13 

Most of the time 71 

Sometimes 


Rarely 


Never 
Not Applicable 


If the ticket was closed, how often is the issue ultimately resolved so that eligibility or benefits can be 
determined? 


Total Responses: 303 Nonresponse: 3 
Always 15 
Most of the time 114 


Sometimes 100 
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Rarely 23 
Never 0 
Not Applicable 51 


If a ticket is closed without resolution, how often is a reason given? 
Total Responses: 303 Nonresponse: 3 

Always 31 

Most of the time 69 

Sometimes 66 

Rarely 41 

Never 21 

Not Applicable 15 

Please select what communication you have received concerning issues you have reported. 

Total Responses: 300 Nonresponse: 6 

Issue or Ticket number 239 

Request/Follow-up for Issue details 118 

Update on issue status 98 

Notification of when the issue will be fixed 99 

Reason for closing the issue (fixed or not) 98 

| don't report issues 24 

Other (please specify) 37 

How often is the issue management process effective and providing a helpful service for users? 
Total Responses: 298 Nonresponse: 8 

Always 10 

Most of the time 66 

Sometimes 

Rarely 

Never 

Not Applicable 


Does your job require you to use information that is either: 1) provided to CHIMES from another system; or 
2) received from CHIMES? 


Total Responses: 462 Nonresponse: 50 

Yes 423 
No 39 
How often do you use these interfaces that provide information to CHIMES? 

Most 

of the 

Total Responses: 409, Nonresponse: 14 Always time Sometimes Rarely Never 

CCUBS Child Care Co-Pay 18 23 32 38 116 
CCUBS Child Care Referral 26 21 25 43 118 
SOLQ SSN, Title 2 & 16 Verification 211 54 16 9 23 
DMS 233 34 17 4 27 
ORION Property Taxes 73 41 60 29 68 
Verify Lawful Presence 54 30 68 52 64 
LIEAP Eligibility 132 64 39 20 37 


Vehicle Title Registration 116 58 66 19 38 





NA 
138 
126 
8/ 
88 
113 
113 
103 
99 


eDRS Disqualification Status 96 39 36 17 62 123 
Other (please specify in 250 characters) 59 


How often do you use these interfaces that receive information from CHIMES? 
Most 
of the 
Total Responses: 409, Nonresponse: 14 Always time Sometimes Rarely Never 


Eligibility Info for SISAWIC 11 9 18 17 156 
Emergency Assistance Info for CAPS 13 16 4‘ 20 136 
TANF Grant Amount for SEARCHS 85 45 44 16 96 
Location Info for SEARCHS 81 42 40 14 97 
Other (please specify in 250 characters) 25 


How often is the information provided in these interfaces useful and accurate? 
Most 
of the 
Total Responses: 409, Nonresponse: 14 Always time Sometimes Rarely Never 


CAPS Foster Care 13 27 25 14 39 
CAPS EA Referral 19 33 24 15 35 
SEARCHS Non-Custodial Parent Info 76 91 53 18 21 
SEARCHS Non-Cooperation Info 72 84 59 19 23 
SEARCHS Out-of-State Info 45 57 51 21 40 
SEARCHS Child Support Payments 94 37 16 22 
CCUBS Income/WoRC/FIA 18 37 33 11 39 
CCUBS Child Care Payments 19 32 27 18 41 
MISTICS Quarterly Wage 92 28 11 15 
MISTICS Unemployment 77 29 15 

BENDEX Income & Disability Info 80 52 19 

Prisoner Verification 62 69 

SSA for SSI and State Supplement Info 33 

SSA SSN and demographic Info 38 


Montana Death Registry 42 
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October 13, 2015 


Tori Hunthausen 


Legislative Auditor OCT 1 3 2095 
Legislative Audit Division REGIS eye aie 
Room 160, State Capitol Building Wsaine EMS UY By 
PO Box 201705 


Helena, Montana 59620-1705 


Re: CHIMES-EA Combined Healthcare Information and Montana Eligibility System-Enterprise 
Architecture 


Dear Ms. Hunthausen: 


The Department of Public Health and Human Services has reviewed the CHIMES-EA Combined 
Healthcare Information and Montana Eligibility System-Enterprise Architecture Audit (1SDP-01) 
completed by the Legislative Audit Division. Our responses and corrective action plans for each 
recommendation are provided below. 


Recommendation #1: We recommend the Department of Public Health and Human Services strengthen 
performance monitoring processes by: 
A. Defining specific goals relative to the system and each program minkaeed by the system. 
B. Defining and documenting a process to consistently review all data relative to these goals. 
C. Reviewing information gathered to identify, document, and track system relation to root cause, 
and communicating results to others. . 


Response: Concur 

Corrective Action: The Department will define consolidated program performance goals and 
document the process to consistently review all data relative to these goals. In addition, the 
Department will continue to use information gathered to identify, document and track system relation 


to root cause and will communicate those results to others. 


Planned Completion Date: 1/31/2016 


Department of Public Health and Human Services 


Director’s Office @ PO Box 4210 Hl Helena, MT 59604 El (406) 444-5622 BI Fax: (406) 444-1970 www.dphhs.mt.gov 


Recommendation #2: We recommend the Department of Public Health and Human Services develop a 
plan to promptly address outstanding system defects that impact eligibility determinations and benefit 
calculations. 


Response: Concur 


Corrective Action: The Department will take a Business Value Assessment of all outstanding 
defects and prioritize them according to criticality of the issue. 


Planned Completion Date: 4/30/2016 


Recommendation #3: We recommend the Department of Public Health and Human Services improve 
interface monitoring by: 
A. Increasing oversight and review of the monitoring process. 
B. Ensuring only necessary information is logged and only users with a business need can access 
logged information. | 


Response: Concur 


Corrective Action: The Department is in the planning and requirements gathering phase of 
implementing continuous monitoring and incident reporting using the Security Information and Event 
Management (SIEM) tool SPLUNK for CHIMES. The Department will add interface monitoring to 
the scope of the SIEM project to provide a more effective method of interface monitoring and 
oversight. The Department has already verified that only necessary information is logged and that no 
PHI or PII is contained in the logs that can be accessed by those without a business need. 


Planned Completion Date: 5/31/2016 
Recommendation #4: We recommend the Department of Public Health and Human Services strengthen 
interface controls by consistently: 


A. Meeting with all stakeholders to document and review current issues. 
B. Communicating with and updating stakeholders throughout the process. 


Response: Concur 
Corrective Action: The Department will meet with and communicate with stakeholders on a more 
regular basis to document and review current issues and will keep them updated throughout the 
process. 
Planned Completion Date: 12/31/2015 
Recommendation #5: We recommend the Department of Public Health and Human Services conduct a 
review of consolidated, statewide overpayment information, including potential overpayments, on a 
periodic basis. 


Response: Concur 


Corrective Action: The Department will conduct a review of consolidated, statewide 
overpayment information, including potential overpayments, on a periodic basis. 


Planned Completion Date: 2/15/2016 


Recommendation #6: We recommend the Department of Public Health and Human Services establish 
and document a process to review metrics related to the source of overpayment claims and address any 
identified functionality issues. 


Response: Concur 


Corrective Action: The Department will establish and document a process to review metrics 
related to the source of overpayment claims and address any identified functionality issues. 


Planned Completion Date: 2/15/2016 


Recommendation #7: We recommend the Department of Public Health and Human Services: 
A. Review claim management processes and controls to continually identify risks and opportunities 
for automated improvements to controls and efficiencies. 
B. Ensure all users are trained appropriately to set up and approve overpayment claims according to 
policy. 


Response: Concur 


Corrective Action: The Department will continue to review overpayment processes to identify if 
there are ways to reduce manual processes. The Department will also evaluate the current training and 
update as necessary to ensure that overpayment claims are established as required by policy and that 
all applicable staff are trained appropriately. 


Planned Completion Date: 12/31/2015 


Recommendation #8: We recommend the Department of Public Health and Human Services ensure all 
current help desk tickets, enhancements, and defects have been prioritized. 


Response: Concur 


Corrective Action: The Department will ensure all current help desk tickets, enhancements and 
defects have been prioritized through our change control process. 


Planned Completion Date: 4/30/2016 


Recommendation #9: We recommend the Department of Public Health and Human Services establish, 
monitor, and enforce benchmarks for: 

A. Analysis and prioritization of help desk tickets, defects, and enhancements. 

B. Time frames for implementing solutions. 


Response: Concur 
Corrective Action: The Department will continue to analyze and prioritize help desk tickets, defects 
and enhancements based on business impact. The future release schedule of monthly and quarterly 


builds, and the summary of the builds completed since the start of 2015, provides timelines for 
implementing solutions. 


Planned Completion Date: 1/31/2016 


Recommendation #10: We recommend the Department of Public Health and Human Services: 
A. Improve the process by ensuring all classifications on help desk tickets are accurate and complete 
and defects are associated as necessary. 
B. Document the process to ensure consistency. 


Response: Concur 


Corrective Action: The Department will ensure all classifications on help desk tickets are accurate 
and complete and defects are associated as necessary and will document the process to ensure 
consistency. 


Planned Completion Date: 4/1/2016 


Recommendation #11: We recommend the Department of Public Health and Human Services establish a 


process to limit reliance on data fixes and ensure data fixes are associated with a long-term application 
fix, if necessary. 


Response: Concur 


Corrective Action: The Department will establish a review process for data fixes and ensure that 
data fixes are associated with a long-term application fixes, only if necessary. Data fixes, that 
may be recurring in nature, will still be done when needed as a temporary resolution to mitigate 
issues while the permanent fix is being prioritized, completed and tested. 


Planned Completion Date: 4/1/2016 


Recommendation #12: We recommend the Department of Public Health and Human Services increase 
communication to users regarding: 

A. Individual help desk ticket resolution and any related defect resolutions. 

B. Defects being addressed through each build and what workarounds will no longer be necessary. 


Response: Concur 


Corrective Action: The Department will incorporate more communication into the release notes 
related to a build, including training documentation that updates users when the need to perform an 
interim solution is no longer needed. Users will continue to receive the resolution of every ticket 
submitted through the service desk process. 


Planned Completion Date: 1/31/2016 


Recommendation #13: We recommend the Department of Public Health and Human Services improve 
current access review procedures by including: 

A. What privileged functions are allowed in each role and if it is necessary. 

B. Which users have roles with access to privileged functions. 


Response: Concur 


Corrective Action: The Department will enhance our process to monitor user access and roles in 
CHIMES EA and improve the documentation related to the review of privileged functions and roles 
as part of our security access reviews. 


Planned Completion Date: 2/15/2016 


Recommendation #14: We recommend the Department of Public Health and Human Services: 
A. Document and track contractor access to system databases. 
B. Monitor contractor activity within system databases. 


Response: Concur 


Corrective Action: The Department will improve the documentation and tracking of contractor 
access to the system databases by enhancing the existing processes for documentation, tracking and 
auditing system access. The Department is in the planning and requirements gathering phase of 
implementing continuous monitoring and incident reporting using the Security Information and Event 
Management (SIEM) tool SPLUNK for CHIMES. The Department will also enhance existing 
database monitoring and oversight of activity within system databases. 


Planned Completion Date: 5/31/2016 


Recommendation #15: We recommend the Department of Public Health and Human Services strengthen 
manual change controls by implementing procedures to monitor manual issuances, manual supplemental 
payments and overrides. 


Response: Concur 


Corrective Action: The Department will monitor the frequency and use of the manual functions 
available within the system to ensure data and program integrity. 


Planned Completion Date: 4/30/2016 


Recommendation #16: We recommend the Department of Public Health and Human Services improve 
user training by: 
A. Covering scenarios that introduce more policy. 
B. Evaluating how satisfied users are with the amount of preparation and training they received. 
C. Update training on a regular basis. 


Response: Concur 


Corrective Action: The Department will continue to review and enhance training experiences and 
resources for staff and is committed to supporting staff competence related to case processing. 


Planned Completion Date: 3/31/2016 


Recommendation #17: We recommend the Department of Public Health and Human Services create or 
update documentation and strengthen controls for reviewing documentation in the following areas: 
Policy referencing previous systems 

Business Rules and Correspondence Detailed Design Documents 

Interface Design Documentation and Data Sharing Agreements 

Benetit Discrepancy Business Rules and Process 

Issue Management Procedures 

System Testing Plans 

User Guides and System Manuals 
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Response: Concur 


Corrective Action: The Department will update policies to remove inappropriate references to 
previous systems. The Department will also update documentation and strengthen controls for 
reviewing documentation in the other areas referenced. 


Planned Completion Date: 6/31/2016 


We appreciate the effort that your staff put into this audit and look forward to using these 
recommendations to continue improving operations and decrease risk within the Medicaid Eligibility 
System. 


Sincerely, 


OX OGL . Rkhecd # Oper 


Richard H. Opper, Director 
Department of Public Health and Human Services 


Ce. Stuart Fuller, Technology Services Division Administrator 
Jamie Palagi, Human and Community Services Division Administrator 
Marie Matthews, Operations Services Branch Manager 
Bob Runkel, Economic Security Services Branch Manager 
Becky Schlauch, Business and Financial Services Division Administrator, Audit Liaison 


